We software testers take a lot for granted.
Our overall assumption is that the only thing that
will fail is the software under test, but
underlying that assumption are a number of other
assumptions about the testing environment –
assumptions about the CPU, the disk drives or
other mass-storage devices, the operating
system, the libraries, our testing framework...
The list goes on and on.
Here is a non-exhaustive list of infrastructure
pieces that could fail while you are testing
your software:
- CPU
- Modern CPUs are a miracle of design and
manufacturing, but they are not error-free.
Usually, the OS and compiler obscure most
CPU defects. This is not always the case,
however – remember Intel's troubles
of years back as immortalized in the phrase:
We are Pentium of Borg. Division is futile.
You will be approximated.
Unless you are writing software to test
CPUs, you can put worrying about whether
your CPU has an error near the bottom
of your list.
- Hard Drives (Mass Storage)
- Mass storage has been dominated by
hard drives for (I think) 50+ years now,
and is likely to continue under that domination
for some time yet, although modern
semiconductor memories
are getting large enough to take over
some of the smaller mass storage applications
(USB keychain "drives", anyone?)
Hard drives are a particular problem,
because their moving parts make them
much more vulnerable to failure.
Hard drives don't always fail catastrophically,
either – occasionally-bad sectors
can make using a hard drive problematic
at best.
- Networking Hardware
- Intermittent network hardware failures
can be devilish to solve, as they just add to
the basic problem of networks – not
all of a network is likely to up at once,
and the larger the network, the more likely
this is (the Internet can be seen as the
Reductio ad absurdum of this line of reasoning, as
there is never a time when all the servers
on the Internet are available).
- Operating System
- All though much current attention is directed towards OS security defects, other OS defects can interfere with your testing just as well. For example, in Microsoft Windows, tests of window message handling can fail because another, errant program is sending messages to your program's window by mistake – a design defect (IMHO) that I suspect is left over from Windows 1.0.
- Compilers
- Each new field in technology requires taming before it can be put to wide use, and compilers are no exception. When Dave Cutler was starting on the architecture of VAX/VMS, compilers were not up to his exacting standards, so the bulk of VAX/VMS (as I understand it) was written in assembly language (yes, despite the previous Unix kernel experience of writing most the OS in a high-level language). Tracking compiler bugs was practically a hobby among some PC programmers during the 1980s, as compilers were still developed mainly by hand, rather than using automated lexers and parser-generators, which introduces more chances for defects to creep in during compiler development. I really can't remember the last time I tracked down an actual compiler bug, but it was probably sometime during the 1990's.
- Libraries
- Libraries now dwarf the size of their associated compilers and interpreters, and the ever-expanding size of these library codebases increases the chance there will be one or more defects in the libraries (see the recent news about how most binary search implementations are likely to be broken, as they run into 32-bit size limitations). One of the beauties of CPAN is that you get the library sources, so
you
can fix problems you might encounter.
- Testing Framework
- Just because you use your testing framework to test your other code doesn't mean your testing framework is defect-free. Perl's Test::Simple and Test::More come with complete sets of automated tests. Recently, over a few lunch hours at work, I wrote VB6Unit, a VB6-based testing framework using the Visual Studio Test System API and plug-in output modules with a reasonably complete automated test suite. Although it is likely that I will find defects in VB6Unit, the automated test suite helps to ensure that the remaining defects are in the edge cases but not the core testing code (Nothing vs. Null vs. Empty, anyone?)
- Security Permissions
- Security permissions are a good example of external factors that can affect your test environment. Unless you use mock objects for all your external interfaces, you have likely had one or more tests not go right because you had a file, database, or network permission set wrong. Servers seem especially vulnerable to security permissions problems, as permissions that are fine for one application (maintained by SysAdmin Foo) are not quite right for another application (maintained by SysAdmin Quux), which leads to problems when Foo and Quux don't think to talk about all their changes in their shared application objects.
That was eight of the myriad ways for your tests to fail even when you have constructed the tests correctly. Do not despair, dear reader – computers and their software are now reliable enough that errors in your tests can first be assumed as defects in your reasoning, not defects in your testing environment. My little guide is just a "heads-up" that once in a while, occasionally, (and usually without warning), your perfectly good testing code will fail you because of a testing infrastructure dependency fault.
